After upgrade or reinstallation, cert-manager does not refresh the mutating or validating of webhook configurations, causing failure to create Issuers and Certificates.

Troubleshooting

Problem

After upgrade or reinstallation, cert-manager does not refresh the mutating or validating of webhook configurations. causing failure to create Issuers and Certificates.

$ERROR controller.certmanager-controller Reconciler error {"name": "default", "namespace": "", "error": "Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": Post \"https://cert-manager-webhook.ibm-common-services.svc:443/mutate?timeout=10s\": x509: certificate signed by unknown authority"}$

Symptom

The following three cases show when the Cert Manager might fail to call webhook.

Case 1

IBM Cert Manager Operator is installed in the cluster, but IBM Common Service Operator did not install cs-ca-certificate, cs-ca-issuer, and cs-ss-issuer resources.

Cert Manager Controller pod shows the following error messages:

2023-04-28T13:40:42.299Z ERROR controller.certificate-controller failed to create v1 Certificate {"name": "ibm-monitoring-certs", 
"namespace": "ibm-common-services", "Request.Namespace": "ibm-common-services", "Request.Name": "ibm-monitoring-certs", 
"error": "Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: 
Post "https://cert-manager-webhook.openshift-cert-manager.svc:443/mutate?timeout=10s\": service "cert-manager-webhook" not found"}

Case 2

After upgrading foundational services to a new version, you are not able to create Issuers and Certificates, and the following error is displayed in the logs:

ERROR   controller.certmanager-controller Reconciler error {"name": "default", "namespace": "", "error": "Internal error occurred: failed calling webhook
"webhook.cert-manager.io\": Post \"https://cert-manager-webhook.ibm-common-services.svc:443/mutate?timeout=10s\": x509: certificate signed by unknown authority"}

Case 3

When you have installed IBM foundational service Cert Manager or CNCF Cert Manager in the cluster before, uninstall them, then try to install IBM Cert Manager, IBM Cert Manager operator pod is running but cert-manager-controller, cert-manager-cainjector, cert-manager-webhook pod are not deployed.

IBM Cert Manager Operator pod shows the following error messages:

2023-05-05T20:22:29.090Z INFO controller_certmanager Failed to create Issuer {"name:": "smoke-check-issuer", "namespace:": "cs-control"}
2023-05-05T20:22:29.090Z INFO controller_certmanager Checking if error is from webhook
2023-05-05T20:22:29.090Z INFO controller_certmanager Auto-detection found error with calling cert-manager-webhook, verify your open source cert-manager installation, and then restart this pod

Case 4

When you have multiple cert-manager installed and delete one of the cert-manager, depending on the sequence of original installation, the mutatingwebhookconfiguration/cert-manager-webhook namespace can be pointing to the wrong namespace, resulting in the following error is displayed in the logs:


E0828 12:48:33.340398 1 controller.go:163] cert-manager/controller/certificates-issuing "msg"="re-queuing item due to error processing" 
"error"="Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: 
Post \"https://cert-manager-webhook.ibm-common-services.svc:443/mutate?timeout=10s\": service \"cert-manager-webhook\" not found" "key"="xxxx/internal-tls-pkcs12-certificate"

Cause

After upgrade or reinstallation, cert-manager does not refresh the mutating or validating of webhook configurations.

Environment

IBM foundational service upgraded to later version

Resolving The Problem

Make sure the cert-manager operator is running , Following are sample from a good environment
oc get pods -A | egrep cert-manager

oc get pods -A | egrep cert-manager 

ibm-common-services   cert-manager-cainjector-cf5bd96f7-fgxxh      1/1     Running  
ibm-common-services   cert-manager-controller-54f5d8869b-jxc27     1/1     Running  
ibm-common-services   cert-manager-webhook-588dd95bbd-lgbwb        1/1     Running  
ibm-common-services   ibm-cert-manager-operator-74f98bcd96-lw7cf   1/1     Running

You must remove the webhook configuration manually.


oc get mutatingwebhookconfigurations | grep cert-manager                    
  cert-manager-webhook   1          13d

oc get validatingwebhookconfigurations | grep cert-manager                
  cert-manager-webhook    1          13d

Deleting the webhook configurations


oc delete mutatingwebhookconfigurations cert-manager-webhook  -n ibm-common-services
oc delete validatingwebhookconfigurations  cert-manager-webhook  -n ibm-common-services

Restarting the cert-manager-webhook pod resolves this issue. If the cert-manager-webhook pod does not exist, restart the ibm-cert-manager-operator pod.

Related Information

Troubleshooting IBM Cloud Pak foundational services install and upgrade

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSRV9V","label":"IBM Cloud Pak foundational services"},"ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"},{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS8QTD","label":"IBM Cloud Pak for Integration"},"ARM Category":[{"code":"a8m0z0000001hogAAA","label":"Common Services"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Tips